Advantage Actor-Critic
Advantage actor-critic (A2C) is an improvement of actor-critic. The introduce of advantage further stablizes the learning procedure of the critic model.
Table of Contents
1. Advantage
The introduction of advantage is important. The reason is that, in a poorly defined enrivonment, even a bad policy could have high rewards. Therefore, directly training by maximizing rewards may falsely encourage actions that should be discouraged.
So, we introduce the advantage function \(A^{\pi}\) to tell how good this aciton is compared to other actions.
\[ A^{\pi}(s,a) = Q^{\pi}(s,a) - V^{\pi}(s) \]
If \(A^{\pi}\gt 0\), then this action is better than average, so we should increase its probability; conversely, if \(A^{\pi} \lt 0\), then this action is worse than the average, we should decrease its probability.
1.1. How to Estimate
The \(A^{\pi}\) here is the accurate value that relies on accurate \(Q^{\pi}\) and \(V^{\pi}\). However, these two are often unknown in practice. We have to estimate \(A^{\pi}\). We use \(\hat{A}_{t}\) to denote the estimate of \(A^{\pi}\) of step \(t\).